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<§) Method and system for associating audio components with positions on a data processing system 
display to produce audio outputs. 



(5?) The present invention associates audio com- 
ponents with positions on a display in a data 
processing system so as to produce audio out- 
puts. The display is configured by providing a 
region definitions or coordinate systems, posi- 
tions on the display using a selected region 
definition, audio components, events that 
trigger the production of audio outputs from the 
defined audio components and associations 
between the audio components, the positions 
and the events. When a defined event occurs in 
a defined position, an audio output is produced 
from the audio component that is associated 
with the defined position. 
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The present invention relates to associating au- 
dio components with visual information in a data proc- 
essing system, and more specifically to associating 
audio components to a display in a data processing 
system so as to produce audio outputs. 5 

In a data processing system, user interfaces al- 
low users to access the data processing system. A 
user interface typically includes a keyboard and a dis- 
play. The display is a visual medium that provides in- 
formation to the user. In addition, the user interface 10 
may include a speaker to provide audio outputs to the 
user. Audio components are used to produce the au- 
dio outputs on the speaker or some other audio out- 
put device and include tones, frequencies, prerecord- 
ed audio and textual strings for use in text-to-speech is 
operations. 

In the prior art, the audio and visual mediums 
have been integrated together by assigning audio 
components to objects presented on the display. For 
example, one way to display information visually is to zo 
use icons. An icon of a printer denotes the hardware 
printer device used by the data processing system. 
The prior art allows an audio component to be as- 
signed to the printer icon on the display, so that when 
the printer icon is utilized in some manner, the audio 
component associated with the printer icon produces 
an audio output to the user through the speaker. In 
addition, the prior art provides the capability of creat- 
ing audio tone changes as a cursor approaches a des- 
ignated object 

The prior art is limited to assigning audio compo- 
nents to objects. As an object is moved to various pos- 
itions on the display, any audio component that is as- 
sociated with the object moves with the object. It is, 
however, desirable to assign audio components to 
positions on a display independently of any objects. 
Such an arrangement would facilitate organization of 
the display by allowing audio components to be asso- 
ciated with defined positions or locations on the dis- 
play. The audio component for the defined position 
would be produced as an output to the audio output 
device whenever a triggering event occurred in the 
defined position. 

In addition, such an arrangement would allow a 
user to configure a display to facilitate navigation of a 
cursor on the display. For example, a visually im- 
paired user might configure a display so that as the 
cursor is moved to a new location on the display, the 
new location is identified by an audio output This type 
of audio feedback would also be helpful to a visually 
impaired user of a computer aided design (CAD) sys- 
tem, where cursor movements are critical in def ining 
geometric shapes. 

It is an object of the present invention to provide 
a method and system for associating audio compo- 
nents with positions on a display of a data processing 
system to produce audio outputs. 

The present invention provides audio outputs that 



are associated with positions on a display means in 
a data processing system. The present invention as- 
sociates a predetermined audio component and a 
predetermined event to one of said positions on the 
display means. An audio output is produced from the 
audio component that is associated with the one pos- 
ition, after the event has occurred in the one position. 

Fig. 1 is a block diagram of a data processing sys- 
tem on which the present invention can be practiced. 

Figs. 2-6 are flow charts showing the method of 
the present invention, in accordance with a preferred 
embodiment Fig. 2 shows the method for managing 
region definitions and defined positions. Fig. 3 shows 
the method for managing defined audio components. 
Fig. 4 shows the method for managing defined 
events. Fig. Sshowsthe method for managing the as- 
sociations between the defined audio components, 
the defined positions and the defined events. Fig. 6 
shows the activation method for executing an audio 
component that is associated with a defined position 
when an associated defined event has occurred. 

Figs. 7-1 0 are schematic views of a computer dis- 
play screen, showing plural icons thereon for illustrat- 
ing the operation of the present invention. 

In Fig. 1 , there is shown a block diagram of a data 
processing system 11 , upon which the present inven- 
tion can be practiced. As may be seen, the data proc- 
essing system 11 may include a network, such as Lo- 
cal Area Network (LAN) 13 which preferably includes 
a plurality of individual user interfaces in the form of 
computers 15. Each computer 15 includes a display 
27 and a speaker 29. The display 27 has a screen 31 
for presenting visual information to a user. The speak- 
er 29 presents audio outputs to the user. Of course, 
those skilled in the art will appreciate that a plurality 
of Interactive Work Stations (IWS) coupled to a host 
processor may be utilized for each such network. 

As is common in such data processing systems, 
each individual computer 1 5 may be coupled to a stor- 
age device 17 and/or a printer/output device 19. One 
or more such storage devices 17 may be utilized, in 
accordance with the method of the present invention, 
to store applications or resource objects which may 
be periodically accessed by any user within the data 
processing system 11. 

Still referring to Fig. 1, it may be seen that the 
data processing system 11 may also include a main 
frame computer 21 , which may be preferably coupled 
to the computers 15 and their respective displays 27 
by means of a communication link 23. The main frame 
computer 21 may also be coupled to a storage device 
25 which may serve as remote storage for the com- 
puters 15 and displays 27. 

The method of the present invention will now be 
described with reference to the flow charts of Figs. 2- 
6. In the flow charts, the following graphical conven- 
tions are observed: a diamond for a test or decision, 
a rectangle for a process or function and a circle for a 
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connector in exiting to or entering from another part 
of the flow chart These conventions are well under- 
stood by those skilled in the art, and the flow charts 
are sufficient to enable one skilled in the art to write 
code in any suitable computer programming lan- 
guage, such as BASIC, PASCAL or C for a computer 
such as the IBM Personal System/2 (PS/2) family of 
computers which supports these languages. 

The method of the present invention includes the 
configuration methods shown in Figs. 2-5 and the ac- 
tivation method of Fig. 6. The configuration methods 
of Fig.2-5 update a configuration table that is used to 
store the necessary parameters for implementing the 
present invention. The configuration table has a list of 
region definitions, a list of positions, a list of audio 
components, a list of events and a list of associations 
between the audio components, the positions and 
the events. The configuration method of Fig. 2 up- 
dates, by adding or editing, the list of region defini- 
tions and the list of positions. The positions are locat- 
ed on the display screen 31, Fig. 1, and are defined 
according to a region definition or coordinate system. 
The configuration method of Fig. 3 updates the list of 
audio components. The audio components are used 
to produce the audio outputs on the speaker or some 
other audio output device and include tones, frequen- 
cies, prerecorded audio and textual strings for use in 
text-to-speech operations. The configuration method 
of Fig. 4 updates the list of events. The events that are 
defined on the list of events are used to trigger the 
production of audio outputs from the audio compo- 
nents associated with a position. The events can in- 
clude asynchronous events, such as icon animation 
or color change, and synchronous events, such as 
moving a mouse cursor or icons. The configuration 
method of Fig. 5 updates the list of associations be- 
tween the audio components, the positbns and the 
events. The activation method of Fig. 6 produces the 
audio output from the audio component that is asso- 
ciated with a position on the display. 

Referring now to Fig 2, the configuration method 
for managing the list of region definitions and the list 
of positions will be described. The method of Fig. 2 is 
invoked when the user selects to update the region 
definitions or the positions for a particular region def- 
inition. The method is first started and initialized, step 
41. Instep 43, the user is shown, on the computer dis- 
play screen 31, Fig. 1, the current region definition 
list The current region definitions are a list of avail- 
able coordinate systems that are used to define pos- 
itions on a display. The user selects a particular re- 
gion definition for updating, step 45. Then, the user 
updates the selected region definition by adding or 
editing, step 47. The region definition list, as well as 
the other lists in the configuration table, could be null, 
as when the configuration table is initially configured. 
If the list is null (i.e. empty), then the user updates the 
list by adding to it Once the update is made, the 



method checks to determine if the new region defini- 
tion is valid, step 49. If the result of step 49 is NO, then 
the method proceeds to step 51 to determine if the 
user wants to continue the update by modifying the 

5 region definition. If the result of step 51 is NO, the 
method stops, step 53. If the result of step 51 is YES, 
then the method loops back to step 47. If the result 
of step 49 is YES, then the user is shown the current 
position list for the selected region definition, step 55. 

10 The user selects a position for update, step 56. Then, 
the user updates the selected position in the selected 
region definition, step 57. The method then checks if 
the updated position is valid, step 59. If NO, the pos- 
ition is not valid, the method proceeds to step 61 to 

15 determine if the user wants to continue the definition 
of positions, step 61 . If the result of step 61 is NO, the 
method stops, step 53. If the result of step 61 is YES, 
the method loops back to step 57. If the result of step 
59 is YES, the defined position is valid, then the con- 

20 figuration table is updated with the region definition 
and the position, step 63. Then, the method stops, 
step 53. 

Referring now to Fig. 3, the configuration method 
for managing audio components will be described. 

25 The method of Fig. 3 is invoked when the user selects 
to update the audio component list of the configura- 
tion table. The method is first started and initialized, 
step 71 . In step 73, the user is shown, on the comput- 
er display screen 31 , the current audio component list 

30 from the configuration table. The user selects a par- 
ticular audio component for updating, step 75. Then, 
the user updates the selected audio component by 
adding or editing, step 77. The method checks to de- 
termine if the audio component is valid, step 79. If the 

35 result of step 79 is NO, then the method proceeds to 
step 81 to determine if the user wants to continue the 
update by modifying the audio component. If the re- 
sult of step 81 is NO, the method stops, step 83. If the 
result of step 81 is YES, then the method loops back 

40 to step 77. If the result of step 79 is YES, then the con- 
figuration table is updated with the audio component, 
step 85. Then, the method stops, step 83. 

Referring now to Fig. 4, the conf igurahbn method 
for managing events will be described. The method of 

45 Fig. 4 is invoked when the user selects to update the 
event list of the configuration table. The method is 
first started and initialized, step 91. In step 93, the 
user is shown, on the computer display screen 31 , the 
current event list from the configuration table. The 

so user selects a particular event for updating, step 95. 
Then, the user updates the event by adding or editing, 
step 97. The method checks to determine if the event 
is valid, step 99. If the result of step 99 is NO, then 
the method proceeds to step 101 to determine if the 

55 user wants to continue the update by modifying the 
event If the result of step 101 is NO, the method 
stops, step 103. If the result of step 101 is YES, then 
the method loops back to step 97. If the result of step 
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99 is YES, then the configuration table is updated 
with the event, step 105. Then, the method stops, 
step 103. 

Referring to Fig. 5, the association manager 
method will be described. The association manager 
method is invoked when the user selects to update 
the list of associations between the audio compo- 
nents, the positions and the events in the configura- 
tion table. The method is started and initialized, step 
111 . The user is shown the list of current audio com- 
ponents from the configuration table, step 113. Then, 
the user selects an audio component to manage, step 
115. The user is shown the current association list, 
from the configuration table, for the selected audio 
component In step 119, the user links the audio com- 
ponent to specific positions and events, which posi- 
tions and events were defined using the methods 
shown in Figs. 2 and 4. Then, in step 121, the list of 
associations is updated. The method then stops, step 
123. 

Referring now to Fig. 6, the activation method will 
be described. The activation method is invoked when- 
ever an associated event occurs within a defined pos- 
ition. The activation method is started and initialized, 
step 131. In step 133, the configuration table, with its 
lists of associations, audio components, positions and 
events, is accessed. Then, in step 135, the audio 
component that is associated with the event that has 
occurred in the defined position is executed. When 
the audio component is executed, the audio output is 
produced through the speaker. The method stops, 
step 137. 

Referring now to Figs. 7-10 examples illustrating 
the operation of the present invention will be ex- 
plained. Figs. 7-10 show schematic representations 
of the computer display screen 31 . In the embodiment 
described herein, the display screen 31 is incorporat- 
ed into a desktop computer 15 (see Fig. 1). However, 
the display screen 31 could be any screen on a user 
interface of a data processing system. 

The display screen 31 is configured using the 
configuration methods of Figs. 2-5. To associate an 
audio component with a position on the display 
screen 31, the following parameters are specified or 
defined: the region definition or coordinate system; 
the position, as defined by the region definition, on 
the display screen; the audio component; the event 
that triggers the playing of that audio component and 
the association between the audio component, the 
position and the event. 

Referring specifically to Fig. 7, there are shown 
various icons, including a phone icon 111, a file cab- 
inet icon 113, an incoming mail icon 115, an outgoing 
mail icon 117 and a printer icon 119. 

The display screen 31 is configured as follows. A 
first audio component is defined as "JOB COMPLET- 
ED" and is associated with a first position 121. The 
first position 121 is defined using a rectangular grid 



coordinate system of 15 rows by 20 columns, (such a 
coordinate system is shown in Fig. 8). The first posi- 
tion 121 is located at squares 13 (row), 3 (column) and 
1 3 (row), 4 (column). The event for triggering the first 

5 audio component of the first position 121 is defined 
as when objects located on the first position produce 
an audio output. As shown in Fig. 7, the printer 119 is 
located on the first position 121. When a print job is 
completed, the printer icon 119 will produce an audi- 

10 ble signal. This event invokes the activation method 
of Fig. 6 which produces the audio output "JOB COM- 
PLETED" through the speaker. 

A second audio component is defined as THIS 
LOCATION OCCUPIED" and is associated with a 

15 second position 123 (see Fig. 7). The second position 
123 is also defined using a rectangular grid coordin- 
ate system, but with smaller squares. Specifically a 
grid of 30 rows by 80 columns (not shown) is used. 
The second position 123 is located at square 4 (row), 

20 75 (column). The event for triggering the second au- 
dio component of the second position 123 is defined 
as when objects located in the second position come 
in contact with one or more objects dragged there. 
When, as shown in Fig. 9, the incoming mail icon 115 

25 is dragged to thef ile cabinet icon 123, the two objects 
come in contact with each other. This event triggers 
the production of the audio output THIS LOCATION 
OCCUPIED" by invoking the activation method of Fig. 
6. 

30 A third audio component is defined as "YOU 
HAVE WORK TO DO" and is associated with a third 
position 125. The third position 125, Fig. 10, is de- 
fined using fractional specifications, namely leftmost, 
center and rightmost thirds of the display screen 31. 

35 The third position 125 is the entire rightmost third of 
the display screen. The event for triggering the third 
audio component of the third position 125 is defined 
as when objects located in the third position change 
color (due to, for example, an asynchronous event), 

40 animate or are selected. Events that trigger produc- 
tion of the audio output "YOU HAVE WORK TO DO" 
include asynchronous notification of the incoming 
mail icon 115, movement of the incoming mail icon 
115, color change of the incoming mail icon 115, flick- 

45 ering of the printer icon 119 as a result of a completed 
job and asynchronous notification of a telephone call, 
movement or color change of the telephone icon 111. 

A fourth, variable position is defined using a rec- 
tangular grid coordinate system of 15 rows by 20 col- 

so umns, as shown in Fig. 8. An audio component is as- 
signed to every square and is defined as "ROW J, 
COLUMN K", with J being the row number and K be- 
ing the column number. The event for triggering the 
audio component is defined as whenever the mouse 

55 cursor 127 is dragged over a square. Thus, when the 
mouse cursor 127 is moved horizontally toward the 
printer icon 119 in Fig. 8, the following audio output 
will be produced, "ROW 3, COLUMN 12", "ROW 3, 
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COLUMN 13", "ROW 3. COLUMN 14" and so on. 

Although the present invention has been descri- 
bed in conjunction with a computer display screen 31 , 
other types of visual media devices could be used. 
For example, radar screens could be used with the 
present invention. In addition, multidimensional visual 
media devices could be used. 

The region definition for defining a position with 
an associated audio component can be a rectangular 
coordinate system, a polar coordinate system, curved 
lines or fractional specifications. Furthermore, math- 
ematical functions can be used to define the region 
definition. 

The audio component can be provided using text- 
to-speech processes on textual strings or prerecord- 
ed audio. If textual strings are used, conversion to au- 
dio can occur when the display screen is configured, 
or if the data processing system operates at a suffi- 
ciently high speed, then conversion can occur in step 
135 of Fig. 6. 



Claims 

1. A method of providing audio outputs that are as- 
sociated with positions on a display means in a 
data processing system, comprising the steps of: 

a) associating a predetermined audio compo- 
nent and a predetermined event with one of 
said positions on said display means; and 

b) producing an audio output from said audio 
component that is associated with said one 
position, after said event has occurred in said 
one position. 

2. The method of claim 1 further comprising the 
step of defining said one position in accordance 
with a defined coordinate system. 

3. The method of claim 1 further comprising the 
step of defining a region definition for the pur- 
pose of identifying said one position on said dis- 
play means. 

4. The method of claim 1 further comprising the 
steps of: 

a) defining said event as a cursor moving be- 
tween two or more of said positions on said 
display means; and 

b) associating said audio component with 
each of said two or more positions so that as 
said cursor is moved between said two or 
more positions the navigation of said cursor 
on said display means is facilitated. 

5. The method of claim 1 wherein said step of asso- 
ciating said event comprises t he step of associating 
an asynchronous event with said one position. 



6. The method of claim 1 wherein said step of asso- 
ciating said event comprises the step of associat- 
ing a synchronous event with said one position. 

5 7. The method of claim 1 wherein said step of asso- 
ciating said audio component comprises the step 
of associating textual strings for use in text-to- 
speech processes. 

10 8. A data processing system for providing audio out- 
puts that are associated with positions on a dis- 
play means in said data processing system, com- 
prising: 

a) means for associating a predetermined au- 
15 dio component and a predetermined event 

with one of said positions on said display 
means; and 

b) means for producing an audio output from 
said audio component that is associated with 

20 said one position, after said event has occur- 

red in said one position. 

9. The data processing system of claim 8, further 
comprising: 

25 a) means for defining said event as a cursor 

moving between two or more of said positions 
on said display means; and 
b) means for associating said audio compo- 
nent with each of said two or more positions 
30 so that as said cursor is moved between said 

two or more positions the navigation of said 
cursor on said display means is facilitated. 

10. The data processing system of claim 8 further 
35 comprising means for defining said one position 

in accordance with a defined coordinate system. 

11. The data processing system of claim 8 further 
comprising means for defining a region definition 

40 for the purpose of identifying said one position on 
said display means. 

12. The data processing system of claim 8 wherein 
said means for associating said event comprises 

45 means for associating an asynchronous event 
with said one position. 

13. The data processing sytem of claim 8 wherein 
said means for associating said event comprises 

so means for associating a synchronous event with 
said one position. 

14. The data processing system of claim 8 wherein 
said means for associating said audio component 

55 comprises means for associating textual strings 

for use in text-to-speech processes. 
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